nvidia_dcgm: do not replace the DCGM that DGX OS already ships - #1394
nvidia_dcgm: do not replace the DCGM that DGX OS already ships#1394100-JM wants to merge 2 commits into
Conversation
On DGX systems install-dgx.yml unconditionally installs the unversioned datacenter-gpu-manager package, which the CUDA repo resolves to DCGM 3.3.9. DGX OS 7.x ships datacenter-gpu-manager-4-cuda13 (4.5.2). The two series conflict, so apt removes the 4.x packages and installs 3.3.9. The play reports ok and nothing stops; the regression only shows up later: dcgmi diag fails with "Detected unsupported Cuda version" (3.3.9 has no CUDA 13 plugin) and dcgmi discovery -l reports 0 NvSwitches on a DGX B300 that has two. The Ubuntu branch already honours dcgm_pkg_name; only the DGX branch hardcodes the name. roles/nvidia-dgx/vars/ubuntu-24.04.yml lists datacenter-gpu-manager-4-cuda13 for DGX OS 7, so the two roles disagree. Skip the install on DGX when any datacenter-gpu-manager package is already present, and use dcgm_pkg_name otherwise. Observed on DGX B300, DGX OS 7.5.0, driver 580.126.20. Signed-off-by: 백지명 <wlaud9805@gmail.com>
dholt
left a comment
dholt
left a comment
There was a problem hiding this comment.
Please make the DGX task validate the specific acceptable DCGM package/version rather than treating any datacenter-gpu-manager* package as sufficient. Handle at least the existing legacy 3.x state, partial package sets, and the fully absent state so each converges to the correct DGX OS package; also ensure the absent-state package name is explicitly DGX-compatible instead of relying on the current unversioned default.
Automated triage review (agent-generated on the maintainer's behalf; a human maintainer decides merges).
Checking for any datacenter-gpu-manager* package was too loose: a host already downgraded to 3.3.9 stayed there, a partial 4.x install was accepted, and the absent case fell back to the unversioned metapackage that caused the downgrade in the first place. Map each DGX OS release to the package it ships (taken from roles/nvidia-dgx/vars) as dcgm_dgx_pkg_map, overridable through dcgm_dgx_pkg_name. On DGX: fail on an unmapped release, remove installed packages from the other DCGM series, then install the mapped package when it is missing. A host already on the right package is left alone, so a second run reports ok. Signed-off-by: 백지명 <wlaud9805@gmail.com>
|
Pushed a follow-up commit to address this. The DGX branch now converges on the package the DGX OS release ships instead of accepting any On DGX the task removes installed packages from the other DCGM series, then installs the mapped package when missing; a host already on the right package is untouched, so a second run is Two points for you to confirm:
|
|
@dholt The requested changes are pushed (see the comment above for details) — ready for another look when you have a moment. |
2 participants
Problem
roles/nvidia_dcgm/tasks/install-dgx.ymlinstalls the unversioneddatacenter-gpu-managerpackage on every DGX. In the CUDA apt repo that name resolves to DCGM 3.3.9. DGX OS 7.x already shipsdatacenter-gpu-manager-4-cuda13(4.5.2). The two series conflict, so apt removes the 4.x packages and installs 3.3.9. The task reportsok, so nothing in the play fails.The regression is only visible afterwards on the node:
The Ubuntu branch (
install-ubuntu.yml) already uses{{ dcgm_pkg_name }}; only the DGX branch hardcodes the name.roles/nvidia-dgx/vars/ubuntu-24.04.ymllistsdatacenter-gpu-manager-4-cuda13for DGX OS 7, so the two roles currently disagree.Reproduced on DGX B300, DGX OS 7.5.0, driver 580.126.20,
install_dcgm: true(default) viaplaybooks/slurm-cluster.yml.Fix
Map each DGX OS release to the DCGM package it ships (
dcgm_dgx_pkg_map, taken fromroles/nvidia-dgx/vars; overridable viadcgm_dgx_pkg_name):datacenter-gpu-managerdatacenter-gpu-managerdatacenter-gpu-manager(current upstream behaviour kept — see note)datacenter-gpu-manager-4-cuda13On DGX the task now: fails clearly on an unmapped release; removes installed packages from the other DCGM series (
datacenter-gpu-managervsdatacenter-gpu-manager-4-*); installs the mapped package when it is missing. A host already on the right package is untouched. Non-DGX branches are unchanged.State → action on DGX OS 7:
-4-cuda13+-4-core(healthy)datacenter-gpu-manager3.3.9 (this bug)datacenter-gpu-manager-4-cuda13-4-coreonly (partial)-4-cuda13-4-cuda13Note on DGX OS 6:
roles/nvidia-dgx/vars/ubuntu-22.04.ymllists no DCGM package, so the map keeps what this role installs there today. If DGX OS 6 has moved to a-4-cuda12package, that row should change; I could not verify it on hardware.Verification
Restored the DGX OS version on the node (
apt-get install datacenter-gpu-manager-4-cuda13=1:4.5.2-1 datacenter-gpu-manager-4-core=1:4.5.2-1):Fabric Manager logs show the same two switches with 74 active links each. With the patched task, a re-run of
slurm-cluster.ymlleaves the 4.5.2 packages in place.